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Background / Context: 

Description of prior research and its intellectual context. 


The Florida Center for Research in Science, Technology, Engineering and Mathematics (FCR- 
STEM) was awarded a grant by the Florida Department of Education to develop a Mathematics 
Formative Assessment System (MFAS) aligned with the Common Core State Standards (CCSS). 
Intended for both teachers and students, formative assessment is a process that provides feedback 
to adjust ongoing teaching and learning to improve students’ achievement of instructional goals. 
It is a process rather than a test, occurring only when the evidence of student thinking is used to 
adjust instruction to address gaps in knowledge related to instructional objectives. Teachers 
practicing formative assessment in the classroom ask students to perform tasks, explain their 
reasoning, and prove their solutions. The evidence collected enables teachers to differentiate 
instruction based on students’ mathematical thinking and reasoning rather than solely on 
incorrect answers. The MFAS-CCSS was designed to provide teachers with tasks and rubrics to 
employ the following five key strategies: (1) assess the student’s level of understanding during 
instruction, (2) identify the student’s specific misconceptions and errors, (3) examine samples of 
student work for further evidence of student understanding, (4) pose additional questions to elicit 
student thinking, and (5) obtain guidance on next steps for instruction. 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 


The current study — the MFAS-CCSS 2012-13 Study — is the third randomized field trial 
conducted on MFAS. The first was a nine- week pilot study in 2010 with K-3 teacher teams 
using tasks aligned with Florida’s Next Generation Sunshine State Standards (NGSSS). The 
second was a semester-long pilot study in 2012 with Grade 2-3 teacher teams using tasks aligned 
with Florida’s NGSSS. Findings for both pilot studies suggested statistically significant positive 
effects on student mathematics where relatively high levels of treatment implementation were 
present, though no detectable group differences were detected on average with either pilot 
sample. The MFAS-CCSS 2012-13 Study was a year-long study with K-l teacher teams using 
tasks aligned with the CCSS. The purpose of the MFAS-CCSS 2012-13 Study was to evaluate 
the effect the MFAS-CCSS has on teachers’ knowledge of elementary mathematics and students’ 
mathematics learning in Grades K-l. The MFAS-CCSS 2012-13 Study made use of intervention 
protocols and measures developed, field tested, and validated in the prior two pilot studies. 

Setting: 

Description of the research location. 

The MFAS-CCSS 2012-13 Study was conducted in three Florida school districts: a medium- 
sized rural district in the north part of the state (District A); a medium-sized suburban district in 
the central part of the state (District B); and a large urban district in the south part of the state 
(District C). Thirty-two schools across the three districts (four in District A, 16 in District B, and 
12 in District C) were recruited for participation. One school in District B withdrew from the 
study, resulting in a sample of 31 schools. 
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Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 


The sample comprised 301 consenting Kindergarten and Grade 1 teachers (Kindergarten n = 

146; Grade 1 n = 155). Parent consent to participate in the study was obtained for 2,317 
Kindergarten and 2,515 Grade 1 students. Teacher teams were composed of three or more 
teachers with a common instructional assignment (i.e., grade level). 

Intervention / Program / Practice: 

Description of the intervention, program, or practice, including details of administration and duration. 

The purpose of the MFAS intervention was to increase a) teacher knowledge of the learning 
goals defined by the mathematics content in the Common Core State Standards, b) teacher use of 
formative assessment to guide instruction, c) teacher collaboration based around evidence of 
students’ mathematical thinking, and d) differentiation of instruction in mathematics based upon 
the instructional needs of individual students. 

Teachers in the treatment group a) completed three introductory professional development 
modules available online through CPALMS, b) implemented MFAS tasks related to the 
standards they were teaching, c) used MFAS rubrics to evaluate student performance and 
differentiate instruction, and d) participated in weekly community of instructional practice (CIP) 
meetings with members of their grade-level team. Facilitated by district lead teachers, the CIP 
meetings focused on implementation of MFAS-CCSS and differentiation of mathematics 
instruction based on the results. Also, principals of treatment schools completed the online 
module for school leaders and committed to support their teachers in implementing the MFAS 
tasks in their classrooms and collaborating within their teams in using the MFAS system. 

Similar to all teachers in Florida, teachers and principals in the control group had access to the 
MFAS professional development modules as well as tasks and rubrics through the CPALMS 
website. However, they were not expected to complete the modules, use the MFAS tasks and 
rubrics to differentiate instruction, or participate in weekly CIP meetings with grade-level teams. 

Research Design: 

Description of the research design. 


Random assignment of schools. Randomization blocked on district and percent Free/Reduced- 
price Lunch (FRPL), resulting in equal proportions of treatment and control schools within each 
district and near equal mean percentage of FRPL across the two conditions. 

Schools were recruited in the late spring and early summer of 2012. In exchange for 
participation, teachers received all materials and training at no cost. The treatment group 
received training on the MFAS-CCSS tasks and rubrics through embedded, on-the-job training, 
including weekly meetings with their communities of instructional practice. As compensation for 
teacher time and effort, each participating teacher, principal, and assistant principal at schools 
assigned to the MFAS condition received a $500 stipend and each participating teacher and 
principal at schools assigned to the non-MFAS condition received a $300 stipend. 
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Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 


Measures. The measures used in this study include the following: 

• Mathematics Knowledge for Teaching (MKT) assessment of teacher knowledge for 
teaching elementary grades mathematics 

• FCR-STEM Observation Protocol for Formative Assessment in Mathematics (OPFAM) 

• FCR-STEM CCSS Student Mathematics Assessments for Kindergarten and Grade 1 

Data collection schedule. All participating teachers completed pre- and post-forms of the MKT, 
in August and May, respectively, of SY 2012-13. A sub-sample of two teachers per grade level 
per school were randomly selected for classroom observation. Trained observers conducted the 
classroom observations the first two weeks of April 2013. The student assessments were 
administered by trained proctors to all participating classrooms in May 2013. Principals were 
interviewed in June 2013. 

Analytic strategy. Mplus software was used to generate teacher theta scores from MKT data 
(using a 2-pl IRT procedure with known item parameters) and factor scores from the OPFAM 
data (using confirmatory factor analysis). All other analyses reported in this research brief were 
conducted using the SPSS MIXED procedure, which allows for cluster correction at the teacher 
and school level. Thus, MKT and OPFAM scores were analyzed with a 2-level model (teachers 
nested within schools) and student mathematics assessment data were analyzed with a 3-level 
model (students nested within teachers nested within schools). All reported impact analyses 
control for school percent FRPL for SY 2011-12. Analyses of the impact of MFAS on MKT 
scores also controlled for MKT scores at pretest. 

Findings / Results: 

Description of the main findings with specific details. 


Teacher mathematics content knowledge. Analyses of the MKT data reveal a positive 
statistically significant effect for MFAS on teacher mathematics content knowledge. Table 1 
relays statistics for the multilevel model. Scale of the MKT post-test theta scores range from - 
3.06 to 2.14, with a mean of -0.42 and standard deviation of 1.02. Moreover the dependent 
variable is in standard deviation units, approximately, rendering the coefficient for MFAS to be 
interpreted as an effect size of 0.26 (p < .05). Thus, controlling for pretest, MFAS teachers 
scored on average a quarter of a standard deviation higher than control teachers on the MKT. 

(please insert Table 1 here) 

Classroom observation. The observation protocol has 36 items distributed among six subscales 
organized according to Wiliam and Thompson’s (2008) formative assessment strategies: a) 
clarifying and sharing learning intentions and criteria for success, b) engineering effective 
classroom discussions and tasks that elicit evidence of learning, c) providing feedback that 
moves learners forward, d) activating students as instructional resources for one another, e) 
activating students as the owners of their own learning, and f) adjusting the instructional plan 
based on formative assessment results when the evidence of learning indicates it is warranted. 
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No statistically significant effects for found for MFAS on any of the OPFAM subscale factor 
scores. 

Student math performance. Analyses of the student mathematics assessment data reveal a 
positive statistically significant effect for MFAS on student mathematics. Table 2 relays statistics 
for the multilevel models for Kindergarten and Grade 1. Scale of the student assessments are Z- 
scored (M = 0, SD = 1) total number correct. A statistically significant positive main effect was 
found for both grade levels: 0.20 ( p < .05) in Kindergarten and 0.24 (p < .01) in Grade 1. 

(please insert Table 2 here) 

Average annual gains in effect size for math nationally-normed tests suggest an expected 
trajectory of 1.14 standard deviation growth from Kindergarten to Grade 1 and an expected 
trajectory of 1.03 standard deviation growth from Grade 1 to Grade 2 (Bloom, Hill, Black, & 
Lipsey, 2008). Using this to contextualize the effects, we can infer that an effect of 0.20 on 
Kindergarten mathematics represent an added gain of nearly a fifth (0.20/1.14 = 0.18) of a year’s 
worth of growth — an additional month and a half of learning — for using MFAS. For Grade 1, we 
can infer that an effect of 0.24 represent an added gain of nearly a fourth (0.24/1.03 = 0.23) of a 
year’s worth of growth — an additional two months of learning — for using MFAS. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 


Effects found for MFAS on teacher math content knowledge and students’ mathematics 
achievement corroborated the results we have found in prior pilot studies and those that have 
been suggested in the research literature. The benefits for teachers to engage in communities of 
instructional practices to discuss student work and analyze formative assessment data for the 
purposes of differentiated instruction are supported in these findings. Further, these findings also 
support that such practices have demonstrable benefits for their students. 

The absence of findings on the classroom observation data is curious, for it stands to reason that 
some change in teacher classroom practice was likely, if the professional development was to 
translate into positive student outcomes. Perhaps, the observation instrument was not sensitive to 
the most salient instructional practices associated with using MFAS. Analyses conducted to date 
are preliminary, whereby subsequent analyses of the observation data may reveal a subset of 
items that constitute an explanatory factor. 

Next steps. Further analyses are warranted, including the inclusion of other possibly important 
covariates, most notably, student-level characteristics. Further, mediation and moderation 
analyses will be conducted to see how these proximal and distal outcomes interrelate. In 
addition, more intensive measurement work is called for, including using IRT methodology to 
score the student assessment data. Lastly, these analyses are based on all consenting students 
who completed the mathematics assessment; however, some of these students did not enroll until 
late in the year, which may artificially deflate the magnitude of the estimated effects. Thus, 
additional analysis that control for factors such as dosage may prove insightful in understanding 
the effectiveness the Math Formative Assessment System. 
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Appendix B. Tables and Figures 

Not included in page count. 


Table 1 

MFAS Effects on Teacher Math Content Knowledge (n = 30 1 ) 


Fixed Effect 

Coefficient 

Standard Error 

p-value 

Intercept 

-0.45 

0.20 

.033 

Teacher-level 
MKT pretest 

0.66 

0.05 

< .001 

School-level 

MFAS 

0.26 

0.12 

.040 

Percent FRPL 

-0.00 

0.00 

.370 

Note. MKT = Math Knowledge for Teaching. MFAS = Math Formative Assessment System. FRPL = 

Free/Reduced-price Lunch. 

Table 2 

MFAS Effects on Kindergarten and Grade 1 Student Mathematics. 


Fixed Effect 

Coefficient 

Standard Error 

p-value 

Intercept 

Kindergarten (n = 2,317) 
0.58 

0.15 

< .001 

School-level 

MFAS 

0.20 

0.09 

.034 

Percent FRPL 

-0.01 

0.00 

< .001 

Intercept 

Grade 1 (n = 2,515) 
0.71 

0.13 

< .001 

School-level 

MFAS 

0.24 

0.08 

.004 

Percent FRPL 

-0.01 

0.00 

< .001 


Note. MFAS = Math Formative Assessment System. FRPL = Free/Reduced-price Lunch. 
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